Method for equitable resource sharing between local and network filesystems

ABSTRACT

Computing resources shared by local and network filesystems may be used to maximum capacity by a single application, or equitably shared by multiple applications under heavy load. A portion of the resources are set aside for local filesystem use and a small number is reserved for each network filesystem. The remaining resources may be used by either local or network filesystems. An enforcement limit puts a ceiling on the number of resources that can be used by a single network filesystem. When the enforcement limit is applied to a filesystem, the number of executing resource acquisition requests from that filesystem is reduced, by completion of the requests, to the reserved number of resources for that filesystem before any more resources acquisition requests from that filesystem are executed.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No.10/620,835 now U.S. Pat. No. 7,827,556, which are incorporated byreference herein.

This application is related to NETWORK FILESYSTEM ASYNCHRONOUS I/OSCHEDULING, filed concurrently by David Chinner and Michael A. Giganteand incorporated by reference herein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is directed to resource sharing in a computingsystem and, more particularly, to how scarce resources are allocated.

2. Description of the Related Art

A network filesystem, such as NFS from Silicon Graphics, Inc. (SGI) ofMountain View, Calif., provides access to files in centralized storageby client computer systems in a client-server operating environment.Such network filesystems generate resource acquisition requests for readand write access to data and to obtain metadata or information about thedata, such as data structure, etc.

The resource acquisition requests are generated by both localfilesystems that access storage connected to the system executing theprocess that needs access to the data, and network filesystems forremote data accessed via a network connected to the system executing theprocess that needs access to the data. Execution of a resourceacquisition request assigns one or more resources of which there are afixed number, such as handles for buffers, pages of memory for storageof actual data, structures required for data caching, etc.Conventionally, the resources are assigned as the resource acquisitionrequests are generated, whether they are for a local filesystem or anetwork filesystem, until a maximum number of resources are in use. Dueto the remoteness of the data to be accessed, it takes longer to executenetwork resource acquisition requests. As a result, it is not uncommonfor a single or a group of network filesystems to monopolize all of theresources and “starve” other filesystems, particularly localfilesystems, from having their resource acquisition requests executed.

Specifically, when multiple read operations are interspersed with heavywrite traffic, a client node may experience file and directory readdelays due to scheduling difficulties. At any time, write requests cangreatly outnumber read and read-ahead requests, and all new requests areplaced at the end of a request queue. Since an application must wait fora file server to read data before it can continue, read throughput isextremely latency sensitive. Use of read-ahead instructions attempts toreduce the latency by queuing the next block before the applicationrequests it. However, the read-ahead requests are conventionally placedon the same first-in, first-out (FIFO) queue as write requests. Throughthe use of a buffer cache, a client node can queue a large number ofwrite requests, but only a small number of read requests at a time. As aresult, an asynchronous request queue may be dominated by writerequests.

Basically, there are three methods for handling a resource shortage: (A)stop using the resource until there are free resources; (B) manageaccesses to the shared resource such that it is always available but mayinvolve waiting until the resource becomes available; or (C) redesignthe system such that it doesn't need the shared resource to function.Solution (A) is inefficient and solution (C) requires lots ofprogramming and in most cases other trade-offs. Therefore, it isdesirable to implement solution (B) by finding an equitable way ofallocating resources that avoids starvation of one or more filesystems,while executing resource acquisition requests efficiently.

SUMMARY OF THE INVENTION

It is an aspect of the present invention to equitably share computingresources between local and network filesystems, particularly as theresources become scarce.

It is another aspect of the present invention to permit a singleapplication, process or filesystem to maximize use of resources whenthere is no other demand for the resources, but to restrict usage of theresources for access to individual filesystem(s) as necessary to permituse of the resources to provide access to other filesystems.

It is a further aspect of the present invention to provide aconfigurable set of control parameters to adjust behavior of a resourcescheduler.

The above aspects can be attained by a method of processing requests toaccess computing resources while restricting processing of resourceacquisition requests when a number of resources in use is within a firstpredetermined amount of a maximum number of available resources. In oneembodiment, the resource acquisition requests include local resourceacquisition requests generated by at least one local filesystem foraccess to local storage and network resource acquisition requestsgenerated by at least one network filesystem for access to remote datavia a network. In this embodiment, the maximum number of availableresources represents the available resources for the network resourceacquisition requests and in addition, a local reserved number of theresources are available for the local resource acquisition requests.

Preferably, an enforcement limit, smaller than the maximum number ofavailable resources by the first predetermined amount, is flexiblyapplied to the network resource acquisition requests as follows. Eachnetwork filesystem has a soft limit for executing the network resourceacquisition requests, and a first network resource acquisition requestis held in a first filesystem queue if execution of the first networkresource acquisition request would cause the enforcement limit to beexceeded and the soft limit for a first network filesystem thatgenerated the first network resource acquisition request has beenexceeded. The first resource acquisition request and any subsequentlyreceived resource acquisition requests for the first network filesystemcontinue to be held until either the executing resource acquisitionrequests for the first network filesystem are below the soft limit orthe first resource acquisition request has been held on the firstfilesystem queue longer than a predetermined time period. Uponcompletion of execution of each of the resource acquisition requests,execution of a longest held resource acquisition request in acorresponding network filesystem queue is initiated if the correspondingnetwork filesystem queue is not empty and the executing resourceacquisition requests generated by a corresponding network filesystem arebelow the soft limit by a second predetermined amount.

Preferably, when a new network resource acquisition request is received,the network resource acquisition requests are flushed if the maximumnumber of available resources are in use. If flushing does not makesufficient resources available, the new network resource acquisitionrequest and any subsequently received network resource acquisitionrequests are held in a global wait queue until the number of resourcesin use is less than the maximum number of available resources. Theflushing of the network resource acquisition requests is repeated eachtime a new network resource acquisition request is received, until thenumber of resources in use is less than the maximum number of availableresources. When a resource acquisition request completes execution, anynetwork resource acquisition requests held in the global wait queue arereleased if the number of resources in use is less than the maximumnumber of available resources.

These together with other aspects and advantages which will besubsequently apparent, reside in the details of construction andoperation as more fully hereinafter described and claimed, referencebeing had to the accompanying drawings forming a part hereof, whereinlike numerals refer to like parts throughout.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system to which the present invention canbe applied.

FIG. 2 is a flowchart of throttling shared resource requests accordingto the present invention.

FIG. 3 is a graphical representation of limit ranges applied to networkand local filesystems according to the present invention.

FIG. 4 is a flowchart of processing performed upon completion of ashared resource request.

FIGS. 5-7 are graphs of resource usage under three different scenarios.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As illustrated in FIG. 1, local computer system 12 executes one or moreapplications 14 which may send data to local disk 16 with a localfilesystem, or client process 18 for a network filesystem. Both localand network filesystems require shared resources 20, such as buffers toaccess data read or written by application(s) 14.

In the case of a resource acquisition request from a local filesystem,one or more shared resources 20 are assigned for use by the localfilesystem, so that the resource acquisition request can be executed.When the data has been accessed, e.g., data has been copied into theshared resource by the application 14 and then written to local disk 16by application 14, the shared resource(s) is/are released back to theavailable pool of resources for use by subsequent resource acquisitionrequests.

In the case of a resource acquisition request from network filesystem(s)18, the process of requesting shared resource(s) 20 is similar to thatof the local filesystem(s) described above. However, once the sharedresource has been assigned, the network filesystem 18 communicate withfilesystem server 22 to access remote data 24. This can require multiplecommunications potentially in multiple stages with a bounded length timedelay between them. When more resource acquisition requests are beinggenerated than there are shared resources 20, the resource acquisitionrequest generated by local filesystem(s) execute more quickly and giveup their turn to a waiting resource acquisition request which is likelyto be a network resource acquisition request, since they take longer toexecute. As a result, local filesystems are often starved of sharedresources 20 by network resource acquisition requests.

A flowchart for processing of a newly generated resource acquisitionrequest according to the invention is illustrated in FIG. 2. First, itis determined 52 whether a maximum number (M) of available resources foruse by network resource acquisition requests from a network filesystem.As illustrated in FIG. 3, the “hard limit” M is less than the totalnumber of available shared resources by a predetermined amount of sharedresources reserved for local filesystems only. If the hard limit (M) hasnot been reached, it is determined 54 whether the number of resources inuse as reached an enforcement limit (E) smaller than the hard limit (M)by another predetermined amount. The manner by which these predeterminedamounts and the “soft limits” discussed below are determined will bedescribed later. If the enforcement limit has not been reached, thenumber of resources in use is incremented 55 by the number of resourcesrequired by the request and the request is executed 56. Depending uponthe system, the number of resources in use may be directly tracked (ifit is common for a single request to use multiple resources), or thenumber of outstanding, i.e., currently executing, resource acquisitionrequests may be tracked as representing the number of resources in use.

When a request from an application occurs, it is necessary to store thedata for the request or from the request reply while the requestexecutes. A buffer handle provides a management structure for keepingtrack of the request and a place to attach a temporary copy of the dataused by the request.

For example, when a read request occurs from either a local or networkfilesystem, a buffer handle must be obtained before pages of memory canbe allocated to store the result of the read request. Once the bufferhandle and the data storage are ready, the read request can be executed.At some later point in time, notification will be received that therequest has completed, and the data requested will be contained in thepages of memory attached to the buffer handle. The application thenreads the data out of the temporary storage into its own buffer andreleases the buffer handle and pages of memory back into their availablepools.

In the case of a write request, once a buffer handle and pages of memoryhave been acquired, the application copies its data into the buffer andexecutes the request. In the case of a synchronous write, theapplication will wait for the request execution to complete andimmediately return the buffer handle and memory to their respectiveavailable pools. In the case of an asynchronous write request, theapplication does not wait for the request to complete; it continuesonward and hence may have many executing asynchronous write requests ata time. Furthermore, the NFS server may not immediately write the datato disk which leaves a buffer handle whose stored data is in anindeterminate state. Hence when the asynchronous write requestcompletes, the buffer handle is left in an “unstable” state in the firststage of an asynchronous write.

When a buffer handle is in the unstable state, a “write synchronization”must be performed on it before it can be released. This involves afurther communication to the server to ensure that the server haswritten the data to disk before the local resource is freed. A “writesynchronization” request can be sent for each unstable buffer or for arange of buffers that hold a contiguous range of data. The “writesynchronization” requests can be issued at any time after theasynchronous write request has completed. The write synchronizationrequest is the second stage of an asynchronous write request, and mayoccur many tens of seconds after the first stage has completed.

As a result of the method used to execute an asynchronous write request,many thousands of asynchronous write requests can be generated andqueued for execution in a very short period of time. As CPUs and memoryare at least two orders of magnitude faster than a disk, asynchronouswrites can be queued very much faster than they can be executed. If leftunbounded, the asynchronous writes will consume all available resourcesand then take an extremely long time (in computer terms) to executethose requests and free up resources.

In comparison, the number of read and metadata requests that can begenerated in the same period of time is substantially less. The reasonfor this is that the number of read and/or metadata requests that can begenerated in a given period of time is greatly limited by the fact thatthe request must be completed (i.e. the data has been retrieved fromdisk) before the next request can be generated.

Hence, it is the asynchronous write requests that must be limited tostop them from dominating the shared resources. It is also far moreimportant to limit the network resource requests as they takesignificantly longer to execute than local requests. Therefore theaccounting mechanism needed to keep track of the network resourceacquisition requests only needs to track the asynchronous write requestsas only this type of request has the potential to dominate the entireshared resource.

As illustrated in FIG. 3, there is also a predetermined number (S) ofresources reserved for each network filesystem for a total of Nresources. As indicated, M minus N resources are available for use byboth local and network filesystems. However, since local resourceacquisition requests execute more quickly, the enforcement limit E ispreferably applied only against network resource acquisition requests.

As illustrated in FIG. 2, if it is determined 54 that the enforcementlimit has been reached, it is determined 58 whether the soft limit (S)has been exceeded for the network filesystem that issued the incomingnetwork resource acquisition request. The soft limit (S) is the numberof resources reserved for that network filesystem (see FIG. 3). If thesoft limit has not been reached, the number of resources in use isincremented 55 and the network resource acquisition request is executed56.

If it is determined 58 that the soft limit for the network filesystemthat issued the incoming network resource acquisition request has beenexceeded, the incoming network resource acquisition request is placed 60in a first-in, first-out (FIFO) soft limit queue for the networkfilesystem. The network resource acquisition request will remain in thesoft limit queue until the executing resource acquisition requests forthe corresponding network filesystem are below the soft limit or thenewly added resource acquisition request has been held in the soft limitqueue longer than a predetermined time period.

If it is determined 52 that the hard limit (M) has been reached, a localflush of the network resource acquisition request is triggered 62. Thepurpose of the local flush is to free up resources used by asynchronouswrites. The local flush results in a scan of the shared resource forbuffers that are “related” (i.e., belong to the same file) and arecurrently in the unstable state. The buffers that are in the unstablestate can be freed up quickly by immediately issuing a writesynchronization request for those buffers. This then returns the buffersthat were unstable back to the free pool much sooner than waiting forthem to be stabilized by a background flush process, and hence mayenable other requests to be processed now that more resources areavailable for use.

After the flush 62 is completed, it is again determined 64 whether themaximum number of resources available for network resource acquisitionrequests are in use. If the flush 62 was successful in releasing enoughresources to drop below the hard limit (M), the previously describedsteps beginning with the determination 54 of whether the enforcementlimit (E) has been reached are performed. If the flush 52 was notsuccessful in releasing sufficient resources, the process places 66 theincoming network resource acquisition request in a global hard limitqueue used for all network resource acquisition requests and waits forthe next incoming network resource acquisition request or completion ofan executing network resource acquisition request, or the passage of apredetermined amount of time.

The operations performed upon completion 80 of a network resourceacquisition request are illustrated in FIG. 4. First, there is anaccounting 82 for completion of the network resource acquisitionrequest. This includes decrementing the number of requests underexecution and, if the resources 20 are handles, upon completion of asynchronous write the handle count is decremented. In the case of anasynchronous write, the handle count is not decremented until the bufferhandle has been stabilized via a write synchronization request.

Next, it is determined 84 whether any network resource acquisitionrequest are held in the global hard limit queue. If not, it isdetermined 86 whether there are any network resource acquisitionrequests in the soft limit queue for a corresponding network filesystem,i.e., the network filesystem that issued the network resourceacquisition request that just completed execution. If it is determined86 that there are network resource acquisition requests held on thecorresponding filesystem soft limit queue, it is determined 88 whetherit is OK to release the longest held resource acquisition request forthe corresponding filesystem. To avoid repeatedly releasing queuednetwork resource acquisition request(s) only to have the request(s) putinto a queue again, preferably the determination 88 of whether torelease the longest held resource acquisition request for thecorresponding filesystem uses a slightly smaller number than thedetermination 58 of whether the soft limit (S) has been reached. This isknown as using “high and low water marks” in performing comparisons. Ifthe number of executing resource acquisition requests for thecorresponding filesystem has dropped below the low water mark, theoldest network resource acquisition request issued by the correspondingnetwork filesystem is released 90 for execution.

If it is determined 84 that there are network resource acquisitionrequests in the global hard limit queue, it is determined 92 whether thenumber of executing resource acquisition requests has droppedsufficiently below the hard limit (M) to release 94 all of the heldresource acquisition requests for possible execution. These requests mayend up on one of the soft limit queues for the network filesystems afterit is determined 86 whether the enforcement limit (E) is being appliedfor that filesystem. Therefore, the determination 92 of whether torelease the global hard limit queue preferably uses a low water markthat is less than the hard limit (M).

Three different scenarios of operation of a computing system accordingto the present invention will be described with reference to FIGS. 5-7.In FIG. 5, a light load of a single network filesystem is illustrated.As the network filesystem issues network resource acquisition requests,the number of executing network resource acquisition requests graduallyincreases until the enforcement limit (E) is reached and the networkresource acquisition requests begin being queued by application. Asillustrated in FIG. 5, the number of executing resource acquisitionrequests may continue to increase due to local network resourceacquisition requests. When the number of executing resource acquisitionrequests drop below the low water mark of the soft limit (S), networkacquisition requests are released until both the soft limit (S) highwater mark and the enforcement limit (E) are exceeded. This is repeateduntil the network filesystem stops issuing network resource acquisitionrequests and there is a gradual decline in the number of executingrequests.

In the example illustrated in FIG. 6, four network filesystems areissuing network resource acquisition requests to produce a medium load.As a result, usage of the shared resources 20 increases faster and maygo higher than in the example illustrated in FIG. 5, because a singlenetwork filesystem can reach the enforcement limit while the othernetwork filesystem have not reached their reserved number or soft limit(S). The other network filesystems are permitted to continue issuingresource acquisition requests until the reserved number is reached ineach, at which point network resource acquisition requests for all fournetwork filesystems will be placed in soft limit queues corresponding tothe network filesystems until the number of executing resourceacquisition requests falls below the low water mark of the correspondingnetwork filesystem's soft limit (S). The drain rate from the peak to thetrough over the time period A in FIG. 6 is substantially the same drainrate over the time period A in FIG. 5. However, since the peak ishigher, the time period A is longer in FIG. 6 than in FIG. 5.

The example illustrated in FIG. 7 is for a heavy load with a largernumber of network filesystems that are able to reach the hard limit (M).As discussed above, the hard limit (M) is the maximum number of sharedresources available for use by network resource acquisition requests andis less than the total number of shared resources by the number ofshared resources available for use by local filesystem(s). In the timeperiods D, the soft limits are enforced on the filesystem(s) that haveexceeded their reserved soft limit(s), but the number of executingresource acquisition requests continue to increase as resourceacquisition requests are received from filesystems below their softlimit, until the hard limit (M) is reached. During time period C,subsequently issued network resource acquisition requests are placed inthe global hard limit queue. If one or more of the network filesystem(s)stop issuing network resource acquisition requests, the number ofexecuting network resource acquisition requests will drop as illustratedfor time periods E. The number can then begin increasing in the secondtime period D illustrated in FIG. 7, if the number of networkfilesystems issuing request increases again.

Preferably, at least the enforcement limit is user configurable. Inaddition, the high and low water marks of the number (M) of resourcesthat may be used by network filesystems and the high and low water marksof the soft limit (S), i.e., the number of resources reserved for eachnetwork filesystem, may also be configurable by a user, such as thesystem administrator.

The present invention has been described with respect to an embodimentoperating on computer equipment manufactured by Silicon Graphics, Inc.However, it applicable to many types of operating systems, not onlythose running variants of UNIX or LINUX, and many types of networkoperating systems. In addition, soft limits could be configured on a perfilesystem basis; the soft limits could be hard limits (i.e., can neverbe exceeded) to enforce strict usage quotas for each filesystem;multiple soft limits with corresponding enforcement limits could be usedto enable more progressive throttling of requests (i.e. S1 is enforcedwhen above E1, S2 is enforced when above E2, etc., where S1>S2>S3>> . .. >Sn and E1<E2<E3< . . . <En<M).

The many features and advantages of the invention are apparent from thedetailed specification and, thus, it is intended by the appended claimsto cover all such features and advantages of the invention that fallwithin the true spirit and scope of the invention. Further, sincenumerous modifications and changes will readily occur to those skilledin the art, it is not desired to limit the invention to the exactconstruction and operation illustrated and described, and accordinglyall suitable modifications and equivalents may be resorted to, fallingwithin the scope of the invention.

What is claimed is:
 1. A method of processing requests to accesscomputing resources, comprising: restricting, without totallysuspending, processing of resource acquisition requests when a number ofresources in use is within a first predetermined amount of a maximumnumber of available resources; wherein the resource acquisition requestsinclude local resource acquisition requests generated by at least onelocal filesystem for access to local storage and network resourceacquisition requests generated by at least one network filesystem foraccess to remote data via a network; wherein restricting processing ofresource acquisition requests applies to network resource acquisitionrequests and not local resource acquisition requests; detecting that anenforcement limit has been reached, the enforcement limit being smallerthan the maximum number of available resources by the firstpredetermined amount; and placing a new network resource acquisitionrequest in an execution queue associated with a network filesystem whenthe enforcement limit has been reached and at least a secondpredetermined number of the resource acquisition requests associatedwith the network filesystem are being processed.
 2. A method as recitedin claim 1, wherein restricting processing of resource acquisitionrequests applies only to network resource acquisition requestsperforming asynchronous writes.
 3. A method as recited in claim 1,wherein: the maximum number of available resources represents theavailable resources for the network resource acquisition requests and,in addition, a local reserved number of the resources are available forthe local resource acquisition requests, and said restricting applies anenforcement limit, smaller than the maximum number of availableresources by the first predetermined amount, to the network resourceacquisition requests.
 4. A method as recited in claim 1, wherein: eachnetwork filesystem has a soft limit of the resources used for executingthe network resource acquisition requests, and said restricting furthercomprises applying an enforcement limit to the network resourceacquisition requests and holding a first network resource acquisitionrequest in a first filesystem queue if execution of the first networkresource acquisition request would cause the enforcement limit to beexceeded and the soft limit for a first network filesystem thatgenerated the first network resource acquisition request has beenexceeded.
 5. A method as recited in claim 4, wherein said holding of thefirst resource acquisition request and any subsequently receivedresource acquisition requests for the first network filesystem iscontinued until at least one of: the executing resource acquisitionrequests for the first network filesystem are below the soft limit, andthe first resource acquisition request has been held on the firstfilesystem queue longer than a predetermined time period.
 6. A method asrecited in claim 1, wherein said method further comprises, uponcompletion of execution of each of the resource acquisition requests,initiating execution of a longest held resource acquisition request in acorresponding network filesystem queue corresponding to a soft limit ofthe resources used for executing the network resource acquisitionrequests if the corresponding network filesystem queue is not empty. 7.A method as recited in claim 6, wherein said initiating execution of thelongest held resource acquisition request in the corresponding networkfilesystem queue is not performed until executing resource acquisitionrequests generated by a corresponding network filesystem are below asoft limit of the resources used for executing the network resourceacquisition requests by a second predetermined amount.
 8. A method asrecited in claim 1, further comprising flushing the network resourceacquisition requests related to a new network resource acquisitionrequest if the maximum number of available resources are in use when thenew network resource acquisition request is received.
 9. A method asrecited in claim 8, further comprising repeating said flushing of thenetwork resource acquisition requests, until the number of resources inuse is less than the maximum number of available resources by at least asecond predetermined amount.
 10. A method as recited in claim 1, furthercomprising holding a new network resource acquisition request and anysubsequently received network resource acquisition requests in a globalwait queue until a number of resources in use is less than a maximumnumber of available resources.
 11. A method as recited in claim 1,further comprising, upon completion of execution of each of the resourceacquisition requests, releasing any new and any subsequently receivednetwork resource acquisition requests in the global wait queue, if anumber of resources in use is less than a maximum number of availableresources by at least a second predetermined amount.
 12. A method asrecited in claim 1, wherein the resources are handles providing accessto data storage for the local and network filesystems.
 13. At least onenon-transitory computer readable medium storing at least one programembodying a method of processing requests to access computing resources,said method comprising: restricting, without totally suspending,processing of resource acquisition requests when a number of resourcesin use is within a first predetermined amount of a maximum number ofavailable resources; wherein the resource acquisition requests includelocal resource acquisition requests generated by at least one localfilesystem for access to local storage and network resource acquisitionrequests generated by at least one network filesystem for access toremote data via a network; wherein restricting processing of resourceacquisition requests applies to network resource acquisition requestsand not local resource acquisition requests; detecting that anenforcement limit has been reached, the enforcement limit being smallerthan the maximum number of available resources by the firstpredetermined amount; and placing a new network resource acquisitionrequest in an execution queue associated with a network filesystem whenthe enforcement limit has been reached and at least a secondpredetermined number of the resource acquisition requests associatedwith the network filesystem are being processed.
 14. At least onenon-transitory computer readable medium as recited in claim 13, whereinrestricting processing of resource acquisition requests applies only tonetwork resource acquisition requests performing asynchronous writes.15. At least one non-transitory computer readable medium as recited inclaim 13, wherein: the maximum number of available resources representsthe available resources for the network resource acquisition requestsand in addition, a local reserved number of the resources are availablefor the local resource acquisition requests, and said restrictingapplies an enforcement limit, smaller than the maximum number ofavailable resources by the first predetermined amount, to the networkresource acquisition requests.
 16. At least one non-transitory computerreadable medium as recited in claim 13, wherein: each network filesystemhas a soft limit of the resources used for executing the networkresource acquisition requests, and said restricting further comprisesapplying an enforcement limit to the network resource acquisitionrequests and holding a first network resource acquisition request in afirst filesystem queue if execution of the first network resourceacquisition request would cause the enforcement limit to be exceeded andthe soft limit for a first network filesystem that generated the firstnetwork resource acquisition request has been exceeded.
 17. At least onenon-transitory computer readable medium as recited in claim 16, whereinsaid holding of the first resource acquisition request and anysubsequently received resource acquisition requests for the firstnetwork filesystem is continued until at least one of: the executingresource acquisition requests for the first network filesystem are belowthe soft limit, and the first resource acquisition request has been heldon the first filesystem queue longer than a predetermined time period.18. At least one non-transitory computer readable medium as recited inclaim 13, further comprising, upon completion of execution of each ofthe resource acquisition requests, initiating execution of a longestheld resource acquisition request in a corresponding network filesystemqueue corresponding to a soft limit of the resources used for executingthe network resource acquisition requests if the corresponding networkfilesystem queue is not empty.
 19. At least one non-transitory computerreadable medium as recited in claim 18, wherein said initiatingexecution of the longest held resource acquisition request in thecorresponding network filesystem queue is not performed until executingresource acquisition requests generated by a corresponding networkfilesystem are below a soft limit of the resources used for executingthe network resource acquisition requests by a second predeterminedamount.
 20. At least one non-transitory computer readable medium asrecited in claim 13, further comprising flushing the network resourceacquisition requests related to a new network resource acquisitionrequest if the maximum number of available resources are in use when thenew network resource acquisition request is received.
 21. At least onenon-transitory computer readable medium as recited in claim 20, furthercomprising repeating said flushing of the network resource acquisitionrequests, until the number of resources in use is less than the maximumnumber of available resources by at least a second predetermined amount.22. At least one non-transitory computer readable medium as recited inclaim 21, further comprising holding a new network resource acquisitionrequest and any subsequently received network resource acquisitionrequests in a global wait queue until a number of resources in use isless than a maximum number of available resources.
 23. At least onenon-transitory computer readable medium as recited in claim 13, furthercomprising, upon completion of execution of each of the resourceacquisition requests, releasing any new and any subsequently receivednetwork resource acquisition requests in the global wait queue, if anumber of resources in use is less than a maximum number of availableresources by at least a second predetermined amount.
 24. At least onenon-transitory computer readable medium as recited in claim 13, whereinthe resources are handles providing access to data storage for the localand network filesystems.
 25. A client node, having a limited number ofresources, in a network of computer system nodes, comprising: at leastone local disk with local filesystems; a communication link to obtaindata accessed under control of another of the computer system nodes; andat least one processor programmed to restrict processing of resourceacquisition requests generated by at least one filesystem when a numberof resources in use is within a first predetermined amount of a maximumnumber of available resources; wherein the resource acquisition requestsinclude local resource acquisition requests generated by at least onelocal filesystem for access to local storage and network resourceacquisition requests generated by at least one network filesystem foraccess to remote data via a network; wherein restricting processing ofresource acquisition requests applies to network resource acquisitionrequests and not local resource acquisition requests; and wherein the atleast one processor is programmed to detect that an enforcement limithas been reached, the enforcement limit being smaller than the maximumnumber of available resources by the first predetermined amount, andplace a new network resource acquisition request in an execution queueassociated with a network filesystem when the enforcement limit has beenreached and at least a second predetermined number of the resourceacquisition requests associated with the network filesystem are beingprocessed.